Summary

Promotes 5 commits from gastown-staging to main. Three independent fix groups plus a developer-facing test procedure:

1. Boot-hydration timeout fix — unblocks /agents/start during container boot hydration and preserves mayor tools on prewarm.
2. GitHub auth correctness — fresh integration tokens instead of stale stored value, plus distinguished failure messages when no token resolves.
3. Logging hygiene — redundant request-logging middleware replaced with per-route Hono-param tagging.
4. Developer tooling — dev-only convoy debug endpoints and a deterministic review-then-land E2E test procedure.

Also lowers TownContainerDO.max_instances from 800 → 500 (as part of commit 1).

Constituent commits

1. Boot hydration + mayor prewarm fix (2ffcef28f, direct push)

Three independent fixes for the startAgentInContainer timeout regression observed after #2974, plus a tighter container-instance cap.

Symptoms. Production logs were filling with two error patterns since the last gastown-staging→main promotion:

[<DOMAIN>] startAgentInContainer: EXCEPTION for agent <UUID>: TimeoutError: The operation was aborted due to timeout
timeout after 6000ms: ensureSDKServer for <agentId>

Root cause. The control server starts accepting requests immediately at boot (main.ts:83), while bootHydration() runs concurrently and serialises every registry agent + the new mayor prewarm through the global sdkServerLock (createKilo reads process.cwd()/process.env). Fresh /agents/start, /refresh-token, and PATCH /agents/:id/model requests queued behind that work and the DO-side AbortSignal.timeout(60s) (resp. REFRESH_AGENT_TIMEOUT_MS=6_000) fired before they ever got the lock.

The mayor prewarm added in #3122 made things worse on two axes:

1. It built KILO_CONFIG_CONTENT from hardcoded model defaults, so the real /agents/start with the user's actual model triggered ensureSDKServer's "config mismatch — evicting prewarmed server" path on every warm restart, doubling lock-holding time on the critical path the prewarm was supposed to speed up.
2. It was missing GASTOWN_AGENT_ROLE, GASTOWN_AGENT_ID, and GASTOWN_TOWN_ID from the prewarm env. kilo serve snapshots process.env at spawn, and plugin/index.ts:66 keys mayor-tool registration off GASTOWN_AGENT_ROLE === 'mayor'. Without those, the prewarmed server booted with no mayor tools, and the cache hit on the next /agents/start handed that defective instance back to the user — manifesting as "mayor tools became unavailable."

Changes

1. Hydration gate (control-server.ts, process-manager.ts)

New awaitHydration() exported from process-manager.ts: a promise that bootHydration replaces on entry and resolves in a finally. Awaited at the top of /agents/start, /refresh-token, and PATCH /agents/:id/model (before any process.env mutation in the model PATCH path so concurrent requests can't race on env writes before holding the SDK lock). Default-resolved at module init so test/dev contexts that never run hydration aren't blocked.

2. Prewarm config matches /agents/start (Town.do.ts, gastown.worker.ts, process-manager.ts)

New getMayorPrewarmContext() on TownDO returns { agentId, model, smallModel, kilocodeToken, organizationId } resolved the same way _ensureMayor resolves them (config.resolveModel(townConfig, null, 'mayor')). The /api/towns/:townId/mayor-id endpoint now returns that whole context so the container builds a KILO_CONFIG_CONTENT byte-identical to what the next /agents/start will send. Falls back to the bare { agentId } shape for back-compat; the container skips prewarm when model/token aren't available rather than building a config that's guaranteed to mismatch.

3. Mayor workdir + plugin env (agent-runner.ts, process-manager.ts)

• Exported ensureMayorWorkspaceForTown(townId) so prewarmMayorSDK materialises the workspace before ensureSDKServer's process.chdir (was throwing ENOENT on cold containers).
• buildPrewarmEnv now mirrors the mayor-shaped subset of buildAgentEnv: GASTOWN_AGENT_ID, GASTOWN_AGENT_ROLE='mayor', GASTOWN_TOWN_ID, KILOCODE_FEATURE='gastown', KILO_TEST_HOME, XDG_DATA_HOME. New end-to-end test intercepts createKilo and asserts those keys are visible to the spawn.

4. wrangler.jsonc

Lowered TownContainerDO.max_instances from 800 → 500 (manual change).

2. Remove manual request logging middleware (#3158, a6cf1029b)

Removes the redundant request-logging middleware in gastown.worker.ts that logged every request twice (-->/<-- via logger.info) — already covered by the per-route instrumented(c, route, handler) AE event wrapper. Replaces the regex-based logger.setTags block with proper per-route tagging using Hono c.req.param() matching for :orgId / :townId / :rigId / :agentId prefixes. Net diff: ~30 deletions + ~25 additions.

Link: #3158

3. Convoy debug endpoints + E2E test procedure (7f9121ffa, direct push)

Adds three dev-only debug endpoints for autonomous convoy testing without going through the mayor LLM:

• GET /debug/towns/:townId/rigs — list rigs in a town
• POST /debug/towns/:townId/sling-convoy — call Town.slingConvoy() directly
• GET /debug/towns/:townId/convoys — list active convoys with progress

Documents the new endpoints and adds a Test C section to services/gastown/docs/e2e-pr-feedback-testing.md with a deterministic procedure for verifying review-then-land convoys end-to-end (sub-bead PRs into the convoy feature branch, then a landing PR into main). Captures known issues observed during verification: container MTU/TLS handshake failures with github.com, 'failed' blockers not gating dependents, and intermittent polecat skipping of sub-PR creation.

4. Fresh integration tokens for GitHub auth (ce15a6fe7, direct push)

resolveGitHubToken previously preferred git_auth.github_token over the platform integration. Since GitHub App installation tokens have a 1h TTL but git_auth.github_token is only written at rig creation (or rare manual refresh), every long-lived town with an integration was handing out an expired token to:

• Polecat/refinery gh CLI (via GH_TOKEN derived from GIT_TOKEN in the container), surfacing as "Failed to log in to github.com using token (GH_TOKEN). The token in GH_TOKEN is invalid."
• The worker-side PR poller (checkPRStatus, checkPRFeedback, mergePR, areThreadsBlocking) — 401 from api.github.com.
• The /refresh-git-token endpoint the container falls back to on auth failure — it returned the same expired token, so the retry just re-failed.

Fix flips priority to github_cli_pat → live integration → stored github_token (last-resort fallback for towns with no integration). Empty-string responses from the integration service now warn and fall back instead of silently failing. Resolves a fresh token at agent dispatch (startAgentInContainer), merge dispatch (startMergeInContainer), and rig setup (setupRigRepoInContainer) before stuffing GIT_TOKEN into envVars. buildContainerConfig now resolves a fresh token before serializing git_auth.github_token into the X-Town-Config header. Adds 6 unit tests covering the priority chain.

5. Distinguish null causes in PR status polling (#3160, 63873e425)

Fixes #3149.

Replace PRStatusResult | null return type with discriminated PRStatusOutcome union in checkPRStatus. Each null cause (no token, HTTP error, invalid response, unrecognized URL, host mismatch) now surfaces a structured PRStatusError with actionable failure messages.

Key changes:

• resolveGitHubToken returns GitHubTokenResolution with resolution chain tracking which sources were tried (back-compat helper resolveGitHubTokenString exists for non-error-aware callers).
• no_token and non-transient HTTP errors (401/403/404) fail the bead immediately (1 strike).
• invalid_response/unrecognized_url/host_mismatch fail after 3 strikes.
• Transient HTTP errors (5xx/429) keep existing 10-strike behavior.
• poll_transient_count and poll_non_transient_count separate counters (replaces the cross-contaminated single poll_null_count); both reset on successful poll.
• failureKind persisted to bead metadata for analytics.
• AE event pr.poll_failed emitted on terminal failure.
• resolveGitHubToken tracks the configured integration source even when GIT_TOKEN_SERVICE binding is missing.

Link: #3160

Verification

• Unit-tested the hydration gate end-to-end with a fetch barrier (asserts awaiters block while bootHydration is in flight, release when it returns).
• Unit-tested the prewarm env shape end-to-end (drives bootHydration with a /mayor-id fetch mock, intercepts createKilo, asserts GASTOWN_AGENT_ID, GASTOWN_AGENT_ROLE='mayor', GASTOWN_TOWN_ID, GASTOWN_CONTAINER_TOKEN, and a non-empty KILO_CONFIG_CONTENT are all visible at spawn time).
• Reviewed the _ensureMayor model-resolution path to confirm resolveModel(townConfig, null, 'mayor') is byte-identical to what /agents/start will send (mayor role ignores rigOverride entirely in config.resolveModel).
• Manual production verification deferred — these changes target a hot path that's hard to reproduce locally; will monitor Sentry / AE mayor.ensure_decision: short_circuit_warm and agent.startup_phase after merge.
• pnpm --filter cloudflare-gastown typecheck passes.
• Unit tests for PR polling: test/unit/pr-poll-errors.test.ts (checkPRStatus, resolveGitHubToken), test/unit/pr-poll-thresholds.test.ts (failureMessageFor, shouldFailImmediately, shouldCountAsTransient).
• Integration test for no_token immediate-fail path: test/integration/pr-poll-errors.test.ts.
• HTTP error scenarios covered by unit tests (mocking fetch is not practical in Cloudflare Workers integration test runtime).

Visual Changes

N/A

Reviewer Notes

• The /api/towns/:townId/mayor-id response shape is back-compat: the container's Zod schema (MayorPrewarmResponse) accepts both the new full-context shape and the legacy { agentId } shape with .passthrough(), and rolls back to "skip prewarm" on missing fields.
• The organizationId fallback chain in buildPrewarmEnv distinguishes undefined (older worker, fall back to process.env) from null (worker authoritatively says "no org") so a stale env-var value can't override an authoritative null.
• The hydration gate is a single global promise — bootHydration is currently single-call from main.ts. If we ever add periodic re-hydration, the resolver capture should move to a local inside bootHydration (called out in code review as a SUGGESTION, deferred).
• Two SUGGESTION-level findings deferred from code review: (a) prewarmMayorSDK warns but doesn't bail on workdir-mismatch (cheap to harden later), (b) one negative-case timing assertion in the new test relies on a 10ms setTimeout (test still validates the positive case deterministically).
• The refresh-git-token.handler.ts change is a caller update for the new GitHubTokenResolution return type (was string | null).
• The wrangler.jsonc max_instances change (800→500) is from the boot hydration commit (2ffcef28f).